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Abstract 

In England, some children have not reached what are considered to be expected levels in literacy and maths by 
the time they move from primary to secondary school. This is more likely for children living in disadvantaged 
areas. One proposal to address this is the provision of summer holiday schools for potentially disadvantaged 
pupils who are reaching the end of their primary schooling. Future Foundations ran a pilot summer school in 
2012. This 4-week programme was intended to reduce summer learning loss, develop children’s skills and 
confidence and perhaps increase parental engagement in their children’s learning as they prepare for school 
transition. The programme provided targeted small-group academic tuition focusing on literacy and numeracy, 
using a scheme of work written by external experts in consultation with local schools, and a diverse programme 
of enrichment activities. The children involved were at Years 5 and 6 in the summer of 2012 (Years 6 and 7 in 
autumn of 2012). This pilot has been successful in demonstrating that the concept is feasible, with some 
suggested improvements, but it has not yet demonstrated that summer schools are effective in improving the 
educational outcomes of disadvantaged children. 

Keywords: summer school, evaluation. Education Endowment Foundation, school transition, literacy, numeracy, 
catch-up 

1. Introduction 

1.1 Background 

In September 2011, the Deputy Prime Minister of the UK announced that £50 million would be made available 
in England for a summer schools programme every year. The scheme was intended to support disadvantaged 
pupils in the transition phase from primary to secondary school. Over 1,700 schools conducted summer school 
programmes in 2012 and over 1,900 schools conducted summer schools sponsored by Department for Education 
in the summer of 2013. 

On May 2012 the government made an announcement of an additional £10 million fund for pupil premium 
children (from families living below the poverty line) who enter secondary school with below Level 4 in literacy 
(https://www.gov.uk/government/news/10-million-to-boost-literacy-for-year-sevens). This was to help children 
struggling in literacy, who might not otherwise be able to access the wider secondary curriculum. 

Schools and other educational organisations were encouraged to develop programmes that could support 
disadvantaged pupils to catch up. The catch-up literacy projects are a set of educational interventions for pupils 
who are struggling to reach the age appropriate levels according to the standard norms in the UK. They are based 
on the existing evidence that struggling pupils entering the secondary school are more likely to continue falling 
behind as compared to the other classmates, and this further leads to disruptive classroom behaviour (Gorard & 
See, 2013). As part of this, in September 2011 the Education Endowment Foundation (EEF) announced its 
intention to fund a pilot of a US-style academic summer school in England. The objectives of this particular 
evaluation were mainly to assess the impact of summer school programmes on the literacy and numeracy 
learning outcomes of pupil. In April 2012, after an open competition run by the EEF with support from Building 
Educated Leaders for Life (BELL), a major US summer-school provider, it was announced that the Future 
Foundations CIC (Future Foundations) would run the pilot in an Academy in NE London. Funding was provided 
for a four-week summer school for 160 pupils reaching the end of their year five or six. The programme was 
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intended to target pupils from disadvantaged backgrounds, who were underperforming at their expected or 
potential levels, and likely to benefit from participation in the programme. The existing evidence for efficacy of 
summer schools from the US is not yet conclusive. One of the reasons for the pilot programme is that there is 
little robust evidence on the efficacy of the BELL approach, especially in the UK. The situation at this stage was 
not yet at the point of equipoise needed as the basis for a full trial. 

1.2 Existing Evidence Base 

Previously, Chaplin and Capizzano (2006) reported a purportedly randomised evaluation of the BELL summer 
programme in Boston and New York. A total of 1,917 pupils applied to the programme, of which only 1,225 
agreed to be part of the study, yet the random allocation to treatment or control was of the original 1,917. The 
report is not very clear on the numbers finally appearing in the treatment and control groups. Of the 1,225 
consenting, 138 were excluded from the study, leaving 1,087. But in the paper, data are only presented for 835 
cases (44% of the original applicants). This means that the study can no longer be regarded as randomised in 
nature. The summer programme involved both maths and reading, but the results are only presented for reading. 
The difference in test scores between treatment and control groups were negligible, and provide no solid 
evidence of a beneficial impact. The report claims that this is because the control group had 16 more days in 
school before the post-test than the treatment group did. This design flaw may indeed be the reason for the lack 
of a more positive result, but the study does not demonstrate a positive result. The report uses statistical 
significance, incorrectly in this context of a non-random sample. The overall effect size of this summer 
programme on reading, not calculated in the report, was only around 0.02. And the performance of the control 
group in vocabulary was actually better than the treatment group. This report therefore provides far from 
convincing evidence of effect, and it was generated by authors with a potential conflict of interest. 

Harvard Family Research Project (2006), clearly also with a link to BELL and so a potential conflict of interest, 
presented a number of evaluations of the BELL summer schools and accelerated learning programme. These 
include BELL (2001, 2002, 2003). The 2002 programme took place in Washington DC, Baltimore, Boston and 
New York, and was intended to raise reading, writing and maths scores for children from 1 st to 7 th grade. Further 
sites were added in 2006. Programmes were either 4 or 6 weeks in duration. The reports are unclear which sites 
were included in each analysis. No improvement is reported for writing. A “significant” gain is reported for 
reading but not maths following the 4 week programme, but the effect size (not reported) is small. The gains 
were higher in both maths and reading for the 6 week programme, although low income children had lower gains 
and in several years actually lost ground. Higher gains were generally reported for the younger year groups. 

See et al. (2012) conducted a rapid review of the evidence on the transition from Year 6 to Year 7 in England, 
and the possibility of pupils catching up in literacy. They came across two further studies concerning the impact 
of summer school programmes. One study of around 2,000 pupils in transition from primary to secondary 
divided them non-randomly into two groups. It found no differential impact on literacy gain scores between the 
groups who attended a 50 hour summer literacy school compared to a control. Both groups demonstrated an 
equivalent decline in scores from pre- to post-test (Sainsbury et al., 1998). Therefore, it seems that the reason for 
any decline over that crucial summer is not to do with whether literacy practice and teaching takes place. It could 
be due to anxiety about changing school, a change in school routine or a different curriculum emphasis. A 
smaller, more recent study from the US involved 331 pupils from grades 1 to 5 in one school (Kim, 2006). Using 
stratification in terms of pre-test reading ability, pupils were randomly allocated to a treatment or delayed 
treatment in a waiting-list design. The treatment involved receiving 10 free books to read during the summer 
vacation, including postcards and letters to stimulate reading. Using self-report, the treatment group read three 
more books, on average, than the control. However, this did not convert to any difference in the literacy scores 
between the groups after the vacation. The number of pupils is quite small in the age range relevant to this paper 
(grade 5) and 52 pupils moved away during the summer (proportionately for each group and stratum). Put 
another way, what these two studies may show is not that summer interventions cannot work, but that it is not 
just about doing something well-meaning and plausible in summer. For example, it may be necessary to have 
some further input rather than just providing books. On the other hand, the first study suggests that simply 
having more “school” over summer does not help either. As far as we can tell from the evidence here, summer 
school programmes have not yet been shown to be effective in improving literacy for pupils in transition. What 
is needed is much firmer evidence. 

An evaluation of summer school impact in England was conducted by NFER in 2013, in which nearly 21,000 
pupils participated (Marting et al., 2013). The study was conducted through a large scale survey in which the 
target group was pupils in schools that conducted a summer school programme and pupils in comparator schools 
that did not participate. The report mainly suggested a positive account of summer school programmes in terms 
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of pupils’ confidence, readiness to attend the secondary school and socialisation. There was no consideration of 
academic progress. 

The 2012 pilot of the BELL approach described in this paper was therefore designed to test the feasibility of 
organising a summer school in a relatively deprived area. In particular, it sought to assess whether there was 
demand for the programme, whether families would support and sustain the programme, and whether 
professional staff would be willing to work during their summer holidays. The pilot also provided an opportunity 
to test the training, curriculum, activities, delivery and management of the programme, so that the programme 
could be improved if the pilot was determined to have demonstrated its feasibility. 

1.3 The Intervention: Summer School 2012 

The summer school took place in a well-appointed secondary school academy, situated in NE London, with a 
large local population of ethnic minority groups living in mixed and social housing. 

Transport was provided for pupils from their home area to the summer school. This was to assist parents and 
ensure a prompt start. A substantial breakfast was provided on site to encourage healthy eating and as a kind of 
incentive. Pupils were divided into eight teaching groups of between 16 and 20 each, with a trained teacher and 
one or more mentors or peer mentors. This grouping was mixed ability involving pupils from year 5 and year 6, 
and took no account of friendship patterns. The groups of children were divided initially on academic 
achievement. Some adjustments were made over the four weeks, and one member of staff suggested that the 
initial allocation had not worked effectively. 

The morning consisted of literacy and numeracy sessions using a mixture of activities, presentation and team 
work. There was a break between the sessions. Then there was a sandwich-based lunch, and an afternoon of 
enrichment and/or enjoyable activities. Sometimes these activities were for the whole intake, sometimes by 
group and sometimes selected individually from a menu. Parents were permitted to observe. Parents or carers 
were asked to arrange transport home, partly to encourage them to come to the summer school during the final 
sessions. 

2. Method 

2.1 Process Evaluation 

This independent evaluation is of the process. It was arranged at short notice and after many decisions about 
implementation of the intervention had already been made. In addition to observation and interviews, the 
evaluators created an estimated effect size of the gain in teacher assessments and other scores based on those 
attending and not attending the summer school. 

The major purpose of the process evaluation was to assist in the formative development of the intervention for a 
subsequent larger evaluation. This evaluation was conducted jointly through co-operation with the staff at Future 
Foundations and the independent evaluators. It required attention to setting up and delivery of the summer school, 
and recording the perceptions of participants, staff and stakeholders. The evaluators observed the training day 
and four days of the summer school, with two researchers on each occasion. This fieldwork included ad hoc 
interviews with the organisers, trainers, staff including mentors, parents, and pupils, plus participant observation 
of activities and lessons, and observation of administrative and “emergency” procedures. 

2.2 Impact Evaluation 

The study also mimicked an outcome evaluation as far as possible. This component provided an estimate of the 
likely effect size relating to attainment, to assist sample size calculations for any future and more substantive 
evaluations. The pilot summative evaluation is based on pupil educational outcomes as assessed by existing test 
scores leading up to, including, and following Key Stage 2. These are considered separately for numeracy 
(maths), and literacy (reading, plus writing for the eventual Year 6 only). Where possible, the analysis has also 
taken into account free school meal eligibility and other pupil background characteristics at an individual level. 
The analysis presents the average pre and post scores in each subject for those participating in the summer 
school and for as many other pupils as possible from the same schools but who did not participate. These are 
converted into a gain score (post minus pre attainment score) for each group. And these are converted into an 
“effect” size by dividing the difference in the gain scores by the standard deviation of the gain score for the 
comparison (not participating) group. The same process is also used to yield a comparison between groups 
involving only pupils eligible for free school meals (FSM), and again to yield a comparison between groups 
consisting of all summer school participants and their “best” individual match in the other group. The best match 
was done in terms of combined pre scores in numeracy and literacy. If more than one pupil in the comparison 
group had the same pre score as a pupil in the treatment group then the best match involved FSM as well. If there 
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had been more multiples the evaluation would have involved ethnicity, SEN and so on. But in fact, prior 
attainment and FSM were all that was needed to create a single “best” match for each treatment pupil. The fourth 
analysis involved only the treatment group. It was a comparison between the reported percentage of attendance 
at the summer school and the individual gain scores for each subject. These were cross-plotted and also 
summarised as Pearson’s R correlation coefficients. This is a modest test of “dosage”. 

It should be noted that a considerable amount of data was missing from both groups. Cases for whom there was 
not at least one complete pair of pre and post scores for at least one subject were deleted. For the remaining cases, 
any isolated pre or post scores in specific subjects were then deleted. 

In addition. Future Foundations collected attendance records for the pupils at summer school, and pupil 
questionnaires on self-confidence, aspirations, and self-ratings of a number of personal and behavioural factors 
such as creativity or leadership. They planned to conduct focus groups with pupils, parents and staff, and to 
present questionnaires to parents and staff. All of these approaches collected evidence on pupil engagement, 
enjoyment and progress, parental engagement, and teacher satisfaction among others. The evaluation was 
therefore a co-operative enterprise. 

3. Results 

3.1 Lessons 

The lesson plans were developed by two expert teachers in literacy and numeracy. A complete set of 4 weeks of 
lessons and materials was developed. During two day initial training the developers explained to the teachers 
how to follow the suggested lesson plans and the activities. The teachers were asked to make adjustment and 
changes according to the need of their teaching groups. The lesson plan developers did not engage with teachers 
after the training sessions. 

In general, the impression from subsequent lesson observations was that the atmosphere was often very like a 
traditional primary school classroom. Teaching was quite formal, and was observed to be variable in quality. 
There were some quite basic pedagogical and factual errors in some classes, apparent lack of interest by teachers, 
and too many broken promises to pupils. On the other hand, it was clear what the intention of the curriculum was 
and on other occasions it seemed to be delivered as envisaged. The mentors were generally impressive, 
hard-working and considerate. But they had a tendency over time either to mimic teacher behaviour or to get 
overly involved in one-to-one disciplinary procedures that entailed absence from the classroom. 

Numeracy lessons 

The classroom for numeracy was set up with standard primary school tables with up to 6 pupils sharing. Each 
table had a laminated aid showing numbers to 100, shapes, fractions, and measurements. This was the only 
material used that was professional looking and large enough for purpose. The other materials used all looked 
‘home-made’. One teacher said at the outset that there would be treats for the winners of each task. But no notes 
were taken. In fact, it was not possible for the teacher to have noticed the winner of some tasks. And by the end 
of lunchtime no message of the winners had been sent to the Future Foundations office tray/box (the system for 
nominations). This was the first indication of a recurring theme where staff promised rewards or activities to the 
pupils and did not seem to carry those promises out. 

There were a number of problems with the lessons. For example, some teachers spent much of their teaching 
time in giving instructions on the tasks or disciplining the pupils. In addressing the class, the teacher would say 
“all eyes on me” and similar phrases, but never waited for compliance nor insisted on a response. The teachers 
created a very school-like atmosphere for numeracy, giving instructions in a loud voice (given the size of the 
room and class). This was rather different from the impression given during the training, and the instruction not 
to sound and act like a teacher in a standard school. 

At the end of each lesson, there was an emphasis on recording only the fun and enjoyable factors. There may be 
good reasons for this, but it will tend to bias any internal evaluation based on such comments. In the second 
week it was clear that a few pupils were not following the lessons and had become uninterested (or the other way 
around). After two weeks the teacher had changed in line with the Future Foundation policy (that most teachers 
objected to). By the final week, there was considerable lack of interest among some pupils and generally the 
same minority of pupils were being uncooperative. Some pupils looked very tired. 

All of the material produced in the classes over four weeks, such as posters, work sheets, pictures and other 
paper based activities, were pasted by mentors and teachers on a big sketch book as a record for parents’ 
observation. The pupils also gave their own reflections. This appeared to be a useful and appropriately 
appreciative way to wrap up the session. The pupils had been at school effectively for four extra weeks while 
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their peers were on holiday, and they had largely been cooperative and enthusiastic. They all deserved 
considerable praise. 

Literacy lessons 

In the early literacy sessions, pupils were divided into groups of 3 to 4. Pupils were free to choose their groups. 
They were given a note-book for writing feedback or reflections on the activity when the teacher asked them to 
do so. Each table had sufficient pens and coloured markers for writing. The lesson plans focused on areas in 
which pupils find difficulty such as sentence structure, punctuation, vocabulary, spellings and comprehension. 
There were several activities to practice reading and writing in each of the lessons. However, the sessions were 
designed to give an extra opportunity for practice rather than improving their approach or understanding of the 
concepts. There was hardly anything new in these lessons that would not have been introduced in their normal 
schools. 

Several times the teachers had to control pupils who were distracting others. The discipline action was to 
intervene once and then to talk to the pupil separately. Such actions generally improved the behaviour. In order 
to call for silence the strategy was that the teacher clapped three times and pupils had to also clap the same way 
to show their response. This strategy was effective and the teacher was relieved from requesting, shouting and 
repeating the same instructions several times. The mentors were assisting the teachers and they were also 
involved in maintaining the discipline in class. A pupil causing some disruption was first warned and then asked 
to have a face-to-face talk out of the classroom. No other special disciplinary issues arose, and the pupils 
appeared happy with their allocated tables and work partners. 

Overall, it was not clear that the administration of the curricula planned for the numeracy and literacy classes 
was consistent across classes, subjects and teachers. It was certain that not all teachers adhered to the same 
teaching methods or used the same materials. In one of the numeracy classes, an experienced and promoted 
teacher was teaching probability seemingly using an approach of his own. 

3.2 Afternoon Activities 

A range of activities was offered in the afternoons. Sometimes there were plenary activities such as the summer 
Olympics, and sometimes pupils were offered a choice, or there was a round robin. On one visit, the choices 
offered for that afternoon were: Judo, drama, cooking, singing, dance, sports, computer use and library. In 
general, pupils were keen to participate, and the mentors were fully involved. 

The shadow summer Olympics was well-thought of, local and topical, and ambitious. However, in operation it 
looked a little cumbersome, and there was little opportunity for pupils to appreciate the efforts and achievements 
of others. The most worrying aspect concerned the inability of some pupils to compete at the same level and so 
to enjoy the event. When an evaluator pointed out a girl with difficulty walking and asked about provision for the 
less able, the organiser simply said “Well they’re not all going to win prizes are they?” Otherwise, the care 
demonstrated by Future Foundations for the pupils was considerable. On one of the hottest days of the year a 
succession of sun block, hats, water and ice lollies appeared, and were distributed efficiently. 

The programme culminated in a graduation ceremony. In the event, this was so popular with parents that there 
was little space in the hall. After a special lunch, the pupils dressed in graduation robes (provided for the day), 
and rehearsed beforehand with all staff. All pupils appeared to participate in at least one activity and all were 
excited to be part of it. 

3.3 Breaks and Meals 

Staffs were on hand to offer sessions such as dance, “freeze”, and football during breaks, with substantial room 
and facilities for play and sports. The atmosphere was generally good. Of course, there was the occasional 
accident or argument but only very occasionally. Free flow water was available at all times, plus hats and sun 
screen when the weather demanded. The children were cared for, and much of this care had obviously been 
pre-planned. 

Meals observed went well, and lunch was delivered efficiently. The meal options were not to all tastes, but it is 
hard to envisage that they ever could be. The meals were appropriate and relatively simple to serve and clear up. 
On the last day, a rather more lavish lunch was provided, including pizza, rice dishes, ice cream and juices. The 
pupils were generally excited and looking forward to their “graduation”. 

3.4 The Views of Pupils and Parents 

The evaluators conducted ad hoc interviews with pupils and found generally positive views. 

Teachers are kind here. In main school teachers are not kind. If someone does something wrong then 
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teachers will start shouting. Here they just give warnings. 

1 like the dance. There are two mentors. They are showing us dance and we are preparing for the street 
dance performance. I am enjoying a lot. 

One of the pupils interviewed said she enjoyed her experience at the school, and when asked what it was she 
enjoyed, she said: 

1 think it gives me an opportunity to learn so that when I go back to Year 6 I would not miss anything. 
Others observed: 

My teachers, I think they are all great because they sometimes they pushed us and helped us like when we 
are stuck they come and help. They make it easier, and more fun. It’s not just plain old work every day, 
it’s also games and get to know it better because you’re having fun while learning. 

Our school, my friend and I we go to the same school and we have the same teachers, basically it’s not 
fun. We don’t play games like here we play games in maths and encourages us to learn. In school we 
don’t have games. 

All of the pupils asked by evaluators said that they would go back to school and tell their friends about their 
experience and usually would recommend it to their friends. One of the pupils commented that she would 
definitely want to come back again and would insist that she be put on the list. 

However, a number of pupils also complained of fighting and bullying going on, and more mentioned feeling let 
down by the lack of trips and activities that they felt they had been offered. A few had ticked a preference for 
activities and not done them. 

Most reported that their parents were supportive: 

Well my parents wanted me to go abroad but they said that this could be big. May be we could go abroad 
next year in Easter holidays. 

My parents want me to go to school because in the summer holiday pupils do not concentrate on work 
and then they don’t want me to sit in front of the laptop. So it is better. 

The evaluators also spoke to parents themselves on several occasions, who were generally very supportive and 
appreciative of the summer school. Of course, since these informal interviews took place on site these responses 
are limited to parents (almost entirely mothers) who turned up to events or to collect their children. Nevertheless, 
the level of enthusiasm is clear: 

He is enjoying it. If he is enjoying it then it’s better than sitting on game. 

My son is so excited that he wakes up early at seven and gets ready himself to come. He said that it was 
fabulous. I am not doing any kind job. 1 think if he comes here then it’s a good holiday for him. It is better 
than staying at home and playing games on computer. My son is an average pupil in the school. Coming 
here is very good for them I think it would keep him active for all these vacations. These activities will 
keep him active. 

It is very near so there is no reason to miss. You can’t say that I can’t do that. It keeps them occupied. I 
mean at the moment I do not work at the moment. I have two kids and keeping them occupied requires a 
lot of money. Constantly taking them to do different activities. So it’s good for my daughter to get out and 
do stuff. Plus they keep their brain going in the morning. A bit of maths and a bit of literacy 

Some parents also made some formative suggestions for future organisation. Some suggested that the school 
provide transport home as well as in the morning, a hot lunch rather than sandwiches and crisps, and more access 
to water or water fountains. One suggested that parents should also be allowed to attend classes, and two that the 
school should have lasted longer. One did not like the mixing of Years 5 and 6 and the range of pre-existing 
progress and ability, and two suggested that the pupils should have homework. 

The most common complaints were that the activities did not live up to billing, and that more, and more varied, 
activities should have been provided. At least 13 parents made an unprompted comment on this. One said that 
the school had been repetitive, and another that their child had “expected to do more things”. This gap between 
what is promised, or thought to have been promised, and what happens needs to be addressed. Several parents 
wrote about “false promises” or “a lot of broken promises”, and this is an issue that the evaluation team also 
noted more widely. At least two parents mentioned fighting and bullying taking place. 
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5.5 The Views of Staff 

Some of the staff interviewed commented that lessons were repetitive, and wanted better lesson plans with 
greater teacher input, more academic topics, and more differentiation of activities. In general, the staffs were not 
happy that their classes were changed over after two weeks. They suggested running likely scenarios on the 
training days (rather than ice-breaking and brain-storming perhaps). One suggested having refresher training at a 
mid-point in the programme “to reinforce routines and expected behaviours, share best practice and 
problem-solve”. They wanted clearer role descriptions, and especially expectations set for the role of mentors in 
behaviour management (or not). It is clear therefore that some staff also noted many of the same potential 
problems as parents and the evaluators. 

The role of the mentors is an interesting one. The course director explained to the evaluators that the teachers and 
mentors were originally planned to be more different: 

Having the mentors and peer mentors and some teachers around in the afternoons is a very valuable thing. 
I may not have foreseen ... the teachers, I think are increasingly having a kind of plan, have a more kind 
of teaching, coaching some of the mentors, so professionally it’s quite a good process for the mentors 
because they learn off the teachers, the teachers help them in the afternoon and that’s proving quite a nice 
relationship because normally you have TAs not being terribly effective often in a lot of schools, whereas 
we have this very different, kind of very active, younger people who seems to be helping kids a lot. And 
the kids aren’t seeing that big a difference between., going teachers are like this and teachers are like that. 
I guess I probably haven’t foreseen that. I thought teachers in the morning and mentors in the afternoon, 
whereas it’s turning into quite a nice professional relationship. 

But teachers were not so clear on this. One teacher commented: 

I think the mentors could not do much as mentor. The role didn’t fit well. You were disciplining them. 
These guys didn’t really get the chance to coach them...They [Future Foundations] needed to hire some 
people from outside agency to manage the afternoon activities. Mentors and peer mentors ought not to 
have been doing the afternoon. 

Apparently no staff had dropped out or failed to show. However, some teachers had been recruited for two weeks 
only and this necessitated some changes in staffing halfway through the programme (the mentors all signed on 
for four weeks). The course director commented: 

1 think all right because afternoons are often different, so they meet lots of different people which is a bit 
like kind of secondary school. So it’s kind of getting them used to their mentor who is like their form 
tutor who they hang around with, but different people coming in and out but their mentors are there to 
remind them that it doesn’t who they are, they need to behave. 

3.6 Summafive Evaluation Results 
The overall figures 

Tables 1 and 2 present the summary results for all eventual Year 7 pupils for whom there is at least one pair of 
pre- and post-intervention results for either maths or reading. Surprisingly, the KS2 fine points score in maths 
and the raw score in reading is lower on average for those pupils not attending the summer school than the other 
pupils from the same schools used as a comparison. This initial difference is not large, but it does raise the 
question whether some of the most disadvantaged pupils did not take part in the summer school, either through 
selection or self-selection. Both groups show gains over the summer period (i.e. there is no discernible summer 
loss, even for those who do not attend the summer school). The gain scores for both groups when converted into 
a standardised “effect” size show very little difference between the groups. The attendees at the summer school 
are slightly ahead in maths and slightly behind in reading, but the differences are small. Given the small size of 
the summer school group, and the scale of missing data, the most appropriate conclusion to be drawn is that there 
is no evidence of a beneficial impact from the summer school for Year 7 pupils on the basis of this comparison. 
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Table 1. Estimated impact of Summer School Programme on Year 7 Maths 


Treatment group 

N 

KS2 points 

September points 

Gain 

Standard deviation 

“Effect” size 

Summer School 

26 

22.0 

23.2 

+1.15 

2.60 

+0.08 

Comparison 

84 

21.0 

21.9 

+0.95 

2.64 

- 


Note: The “effect” size is the difference between the mean gain scores, divided by the standard deviation of the 
comparison group. 


Table 2. Estimated impact of Summer School Programme on Year 7 Reading 


Treatment group 

N 

KS2 raw score 

September raw score 

Gain 

Standard deviation “Effect” size 

Summer School 

34 

33.4 

34.9 

+1.53 

5.64 -0.02 

Comparison 

53 

32.5 

34.2 

+1.74 

9.70 


Tables 3 to 5 present the summary results for all eventual Year 6 pupils for whom there is at least one pair of pre 
and post results for maths, reading or writing. The initial scores in all three subjects are similar for those pupils 
attending the summer school and the other pupils from the same schools used as a comparison. The initial 
difference is in favour of the comparison group in maths and writing, and in favour of the summer school in 
reading. As with Year 7, there is no clear evidence from this data that the summer school catered for an 
especially disadvantaged set of pupils from these schools. 

Both groups show a decrease in scores over the summer period in all three subjects (i.e. there is a discernible 
summer loss, even for those who attended the summer school). The gain scores for both groups when converted 
into a standardised “effect” size show a negative difference for those attending the summer school for all three 
subjects. This difference is over one quarter of a standard deviation in writing. Given the small size of the 
summer school group for whom scores were provided, and the pilot nature of the intervention, this is not 
definitive evidence of a harmful impact from attending the summer school, but it cannot be construed as 
evidence of any beneficial impact for Year 6 pupils. 


Table 3. Estimated impact of Summer School Programme on Year 6 Maths 


Treatment group 

N 

August score 

September score 

Gain 

Standard deviation “Effect” size 

Summer School 

22 

21.7 

18.7 

-2.95 

2.24 -0.15 

Comparison 

34 

22.1 

19.5 

-2.56 

2.66 


Table 4. Estimated impact of Summer School Programme on Year 6 Reading 


Treatment group 

N 

August score 

September score 

Gain 

Standard deviation “Effect” size 

Summer School 

22 

22.0 

20.1 

-1.86 

3.73 -0.14 

Comparison 

33 

21.6 

20.3 

-1.36 

3.59 


Table 5. Estimated impact of Summer School Programme on Year 6 Writing 


Treatment group 

N 

August score 

September score 

Gain 

Standard deviation “Effect” size 

Summer School 

19 

19.7 

18.8 

-1.57 

2.70 -0.27 

Comparison 

30 

20.6 

19.0 

-0.89 

2.51 


FSM pupils only 

The tables above presented effect sizes based on all pupils for whom there was at least one pair of scores. Given 
that the intervention was intended to provide assistance primarily to pupils from disadvantaged backgrounds, it is 
interesting to consider the scores only for those pupils eligible for free school meals. This introduces a further 
variable with some missing values which means that the number of usable cases drops again and may lead to 
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further selection bias. Again, the initial scores of both groups are close with no evidence that the summer school 
recruited especially poorly-performing pupils from the primary schools involved. In fact, the summer school 
group were already ahead in Year 7 maths and Year 6 reading. Both groups showed an increase over summer in 
Year 7, except for the comparison group in reading which also had a very high standard deviation (Tables 6 and 
7). As above, both groups showed a decrease in scores over the summer in Year 6 for all subjects (Tables 8 to 
10 ). 

There is a small effect size in favour of the summer school in both subjects for Year 7. However, the number of 
cases is even smaller than above. Therefore, it is hard to present this as clear evidence of a beneficial impact 
from the summer school. 


Table 6. Estimated impact of Summer School Programme on Year 7 Maths 


Treatment group 

N 

KS2 points 

September points 

Gain 

Standard deviation 

“Effect” size 

Summer School 

13 

21.9 

22.9 

+0.92 

2.90 

+0.12 

Comparison 

36 

20.3 

21.0 

+0.67 

2.08 

- 


Table 7. Estimated impact of Summer School Programme on Year 7 Reading 


Treatment group 

N 

KS2 raw score 

September raw score 

Gain 

Standard deviation 

“Effect” size 

Summer School 

23 

33.0 

34.4 

+1.48 

6.28 

+0.18 

Comparison 

20 

33.3 

32.6 

-0.65 

11.66 

- 


There is a small negative effect size for Year 6 maths, a small-to-medium size negative effect size for reading, 
and a medium size positive effect size for writing. Again there is no evidence, overall, of a beneficial impact 
from attending the summer school. However, the larger difference in writing is worthy of note, and could be a 
clue to where the impact, if there is any, might lie. 


Table 8. Estimated impact of Summer School Programme on Year 6 Maths 


Treatment group 

N 

August score 

September score 

Gain 

Standard deviation 

“Effect” size 

Summer School 

15 

22.0 

19.0 

-3.0 


2.54 

-0.09 

Comparison 

30 

22.1 

19.3 

-2.77 


2.43 

- 

Table 9. Estimated impact of Summer School Programme on Year 6 Reading 

Treatment group 

N 

August score 

September score 

Gain 

Standard deviation 

“Effect” size 

Summer School 

14 

23.0 

20.9 

-2.14 


4.40 

-0.27 

Comparison 

29 

21.7 

20.5 

-1.21 


3.47 

- 

Table 10. Estimated impact of Summer School Programme on Year 6 Writing 

Treatment group 

N 

August score 

September score 

Gain 

Standard deviation 

“Effect” size 

Summer School 

11 

19.7 

19.5 

-0.27 


2.53 

+0.46 

Comparison 

27 

20.6 

19.1 

-1.56 


2.79 

- 


4. Discussion 

Overall, as a test of concept the summer school must be considered a success. Future Foundations set it up in a 
relatively short period of time, selected a good site, appointed and trained the staff, and populated the school 
with children from the end of Years 5 and 6, many eligible for free schools meals. The target of 160 pupils was 
nearly met, and attendance was largely maintained. Staff, pupils and parents supported the programme and were 
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generally appreciative of the opportunity. The mentors and peer mentors were a noticeable success, whatever 
role they played. Health and safety, provision of food and access to water, and general care were seen to be good. 
The correct number of sessions was staffed at the planned staff: pupil ratios. A range of activities was undertaken, 
culminating in a plenary graduation event. There were no major problems. It seems that mixed summer provision 
for struggling pupils in a relatively disadvantaged area is feasible. 

There is little evidence of pupil progress in attainment in the term following the summer programme. This is 
perhaps largely due to the small scale of the pilot, the considerable amount of missing data, and lack of bespoke 
or focused pre and post data. However, as this report describes, not all lesson delivery was of a high quality even 
though staff sometimes had Advanced Teacher Status (ATS). It is even possible that some did not take the 
summer programme as seriously as their main job. The use of technology was generally poor, and the lesson aids 
(with the exception of the laminated table charts) were often small, hard to read, and looked rushed. The lesson 
plans and pedagogy for literacy and numeracy are key to the success of such a venture, however well other 
activities are organised. In general, also, the light, fun and non-school atmosphere observable outside lessons and 
urged in the staff training were not usually evident in classrooms. Lessons seemed very much like rather ordinary 
school lessons with a slight twist rather than anything more radical. Possible improvements for future rounds 
include the selection and motivation of staff, the use of scenarios in training and perhaps even teaching mock 
classes to each other, and some refresher training mid-way through the 4 weeks. Perhaps the staff selected could 
include educators from other phases or types of learning organisation to try and offer news visions of teaching, 
and to try and break the school-like atmosphere in classes. It is clear that staff generally did not like changing 
classes after two weeks but this lack of popularity does not make it wrong and such decisions must remain in the 
hands of the management. 

The role of the mentors or peer mentors was somewhat unclear. Over the 4 weeks, they tended to ‘mimic’ the 
behaviour of teachers (their own role models in the classroom), or to be focused on discipline issues only. In one 
lesson observed, the mentor took a misbehaving pupil outside early on and never returned. The mentors may 
need separate training from the teachers for more of the time, and their purpose needs to be clearer. They were 
observed at their best in running activities (a role that some staff thought they should not have), and in aiding the 
teacher when dealing substantively (not with discipline) with individuals or tables in class. Not all of these roles 
were dealt with in the training, insofar as it was observed. 

In plenary sessions, classes, and individual interactions the evaluators observed pupils being offered or promised 
things that never materialised. In some cases, such promises were not necessary or particularly important. But 
where they are made they ought to be kept. Pupils and parents will be tolerant of unavoidable problems (such as 
where a planned activity is cancelled due to weather, for example) but if the problem was as endemic as these 
observations and participant comments suggest then it will affect the culture of trust. 

Some of the afternoon activities were more successful and more popular than others. The graduation, raft 
building, games during break and others were appreciated and should be continued, even though they may not 
directly relate to gains in attainment. They are part of the approach adopted. Similarly, some pupils greatly 
appreciated the quieter and less dramatic chance to read a book or tend some plants, for example. Other activities 
were well-meaning but not always implemented well. The summer Olympics appeared to be a missed 
opportunity. All pupils must feel that they can contribute points to their team’s total and so on. Like the lessons, 
such topical or local events can be retained but should be improved. The martial arts session was popular with 
some but clearly a possible source of risk. 

The available figures do not suggest that attendees at the summer school 2012 were particularly under-achieving 
compared to their peers in the same school. Given greater time in any future rounds, it is important to address 
this and ensure that the summer school serves those most in need. 

As stated above, this pilot has been successful in demonstrating that the concept is feasible. It is therefore 
appropriate to move to the next stage of research and development which should consist of a fairer test of the 
summative impact of the programme on literacy, numeracy, school transfer, and pupil attitudes or behaviour. 
This pilot has not demonstrated that the programme is effective in raising attainment. It was not intended to do 
so. Therefore, this evaluation provides no basis for rolling out this kind of intervention more widely at present, if 
the main purpose is to improve literacy and numeracy. 
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